<div dir="ltr">Status, as I understand it:<div><br></div><div>* oslo.db changes to support other mysql drivers:<br></div><div><br></div><div><a href="https://review.openstack.org/#/c/104425/" target="_blank">https://review.openstack.org/#/c/104425/</a> (merged)<br>
</div><div><a href="https://review.openstack.org/#/c/106928/" target="_blank">https://review.openstack.org/#/c/106928/</a> (awaiting oslo.db review)<br>
</div><div><a href="https://review.openstack.org/#/c/107221/" target="_blank">https://review.openstack.org/#/c/107221/</a> (awaiting oslo.db review)<br></div><div><br></div><div>(These are sufficient to allow operators to switch connection strings and use mysqlconnector. The rest is all for our testing environment)</div>
<div><br></div><div>* oslo.db change to allow testing with other mysql drivers:</div>
<div><br></div><div><a href="https://review.openstack.org/#/c/104428/" target="_blank">https://review.openstack.org/#/c/104428/</a> (awaiting oslo.db review)<br></div><div><a href="https://review.openstack.org/#/c/104447/" target="_blank">https://review.openstack.org/#/c/104447/</a> (awaiting oslo.db review. Ongoing discussion towards a larger rewrite of oslo.db testing instead)<br>
</div><div><br></div><div>* Integration into jenkins environment:</div><div><br></div><div>Blocked on getting Oracle to distribute mysql-connector via pypi.</div><div>Ihar and others are having conversations with the upstream author.</div>
<div><br></div><div>* Devstack change to switch to mysqlconnector for neutron:</div><div><br></div><div><a href="https://review.openstack.org/#/c/105209/">https://review.openstack.org/#/c/105209/</a> (marked wip)<br></div>
<div>Ihar: do you want me to pick this up, or are you going to continue it once some of the above has settled?</div><div><br></div><div>* oslo.db gate test that reproduces the deadlock with eventlet:</div><div><br></div><div>
<a href="https://review.openstack.org/#/c/104436/">https://review.openstack.org/#/c/104436/</a> (In review. Can't be submitted until gate environment is switched to mysqlconnector)<br></div><div><br></div><div><br></div>
<div>Overall I'm not happy with the rate of change - but we're getting there. I look forward to getting this fixed :/</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 18 July 2014 21:45, Ihar Hrachyshka <span dir="ltr"><<a href="mailto:ihrachys@redhat.com" target="_blank">ihrachys@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="">-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA512<br>
<br>
</div><div><div class="h5">On 14/07/14 17:03, Ihar Hrachyshka wrote:<br>
> On 14/07/14 15:54, Clark Boylan wrote:<br>
>> On Sun, Jul 13, 2014 at 9:20 AM, Ihar Hrachyshka<br>
>> <<a href="mailto:ihrachys@redhat.com">ihrachys@redhat.com</a>> wrote: On 11/07/14 19:20, Clark Boylan<br>
>> wrote:<br>
>>>>> Before we get too far ahead of ourselves mysql-connector<br>
>>>>> is not hosted on pypi. Instead it is an external package<br>
>>>>> link. We recently managed to remove all packages that are<br>
>>>>> hosted as external package links from openstack and will<br>
>>>>> not add new ones in. Before we can use mysql-connector in<br>
>>>>> the gate oracle will need to publish mysql-connector on<br>
>>>>> pypi properly.<br>
><br>
>> There is misunderstanding in our community on how we deploy db<br>
>> client modules. No project actually depends on any of them. We<br>
>> assume deployers will install the proper one and configure<br>
>> 'connection' string to use it. In case of devstack, we install<br>
>> the appropriate package from distribution packages, not pip.<br>
><br>
>>> Correct, but for all of the other test suites (unittests) we<br>
>>> install the db clients via pip because tox runs them and<br>
>>> virtualenvs allowing site packages cause too many problems. See<br>
>>><br>
>>> <a href="https://git.openstack.org/cgit/openstack/nova/tree/test-requirements.txt#n8" target="_blank">https://git.openstack.org/cgit/openstack/nova/tree/test-requirements.txt#n8</a>.<br>
>>><br>
>>><br>
><br>
>>><br>
So we do actually depend on these things being pip installable.<br>
>>> Basically this allows devs to run `tox` and it works.<br>
><br>
> Roger that, and thanks for clarification. I'm trying to reach the<br>
> author and the maintainer of mysqlconnector-python to see whether<br>
> I'll be able to convince him to publish the packages on<br>
> <a href="http://pypi.python.org" target="_blank">pypi.python.org</a>.<br>
><br>
<br>
</div></div>I've reached the maintainer of the module, he told me he is currently<br>
working on uploading releases directly to <a href="http://pypi.python.org" target="_blank">pypi.python.org</a>.<br>
<div><div class="h5"><br>
><br>
>>> I would argue that we should have devstack install via pip too<br>
>>> for consistency, but that is a different issue (it is already<br>
>>> installing all of the other python dependencies this way so<br>
>>> why special case?).<br>
><br>
>> What we do is recommending a module for our users in our<br>
>> documentation.<br>
><br>
>> That said, I assume the gate is a non-issue. Correct?<br>
><br>
>>>>><br>
>>>>> That said there is at least one other pure python<br>
>>>>> alternative, PyMySQL. PyMySQL supports py3k and pypy. We<br>
>>>>> should look at using PyMySQL instead if we want to start<br>
>>>>> with a reasonable path to getting this in the gate.<br>
><br>
>> MySQL Connector supports py3k too (not sure about pypy though).<br>
><br>
>>>>><br>
>>>>> Clark<br>
>>>>><br>
>>>>> On Fri, Jul 11, 2014 at 10:07 AM, Miguel Angel Ajo Pelayo<br>
>>>>> <<a href="mailto:mangelajo@redhat.com">mangelajo@redhat.com</a>> wrote:<br>
>>>>>> +1 here too,<br>
>>>>>><br>
>>>>>> Amazed with the performance gains, x2.4 seems a lot, and<br>
>>>>>> we'd get rid of deadlocks.<br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>> ----- Original Message -----<br>
>>>>>>> +1<br>
>>>>>>><br>
>>>>>>> I'm pretty excited about the possibilities here. I've<br>
>>>>>>> had this mysqldb/eventlet contention in the back of my<br>
>>>>>>> mind for some time now. I'm glad to see some work<br>
>>>>>>> being done in this area.<br>
>>>>>>><br>
>>>>>>> Carl<br>
>>>>>>><br>
>>>>>>> On Fri, Jul 11, 2014 at 7:04 AM, Ihar Hrachyshka<br>
>>>>>>> <<a href="mailto:ihrachys@redhat.com">ihrachys@redhat.com</a>> wrote:<br>
>>>>> On 09/07/14 13:17, Ihar Hrachyshka wrote:<br>
>>>>>>>>>> Hi all,<br>
>>>>>>>>>><br>
>>>>>>>>>> Multiple projects are suffering from db lock<br>
>>>>>>>>>> timeouts due to deadlocks deep in mysqldb<br>
>>>>>>>>>> library that we use to interact with mysql<br>
>>>>>>>>>> servers. In essence, the problem is due to<br>
>>>>>>>>>> missing eventlet support in mysqldb module,<br>
>>>>>>>>>> meaning when a db lock is encountered, the<br>
>>>>>>>>>> library does not yield to the next green thread,<br>
>>>>>>>>>> allowing other threads to eventually unlock the<br>
>>>>>>>>>> grabbed lock, and instead it just blocks the main<br>
>>>>>>>>>> thread, that eventually raises timeout exception<br>
>>>>>>>>>> (OperationalError).<br>
>>>>>>>>>><br>
>>>>>>>>>> The failed operation is not retried, leaving<br>
>>>>>>>>>> failing request not served. In Nova, there is a<br>
>>>>>>>>>> special retry mechanism for deadlocks, though I<br>
>>>>>>>>>> think it's more a hack than a proper fix.<br>
>>>>>>>>>><br>
>>>>>>>>>> Neutron is one of the projects that suffer from<br>
>>>>>>>>>> those timeout errors a lot. Partly it's due to<br>
>>>>>>>>>> lack of discipline in how we do nested calls in<br>
>>>>>>>>>> l3_db and ml2_plugin code, but that's not<br>
>>>>>>>>>> something to change in foreseeable future, so we<br>
>>>>>>>>>> need to find another solution that is applicable<br>
>>>>>>>>>> for Juno. Ideally, the solution should be<br>
>>>>>>>>>> applicable for Icehouse too to allow distributors<br>
>>>>>>>>>> to resolve existing deadlocks without waiting for<br>
>>>>>>>>>> Juno.<br>
>>>>>>>>>><br>
>>>>>>>>>> We've had several discussions and attempts to<br>
>>>>>>>>>> introduce a solution to the problem. Thanks to<br>
>>>>>>>>>> oslo.db guys, we now have more or less clear<br>
>>>>>>>>>> view on the cause of the failures and how to<br>
>>>>>>>>>> easily fix them. The solution is to switch<br>
>>>>>>>>>> mysqldb to something eventlet aware. The best<br>
>>>>>>>>>> candidate is probably MySQL Connector module that<br>
>>>>>>>>>> is an official MySQL client for Python and that<br>
>>>>>>>>>> shows some (preliminary) good results in terms<br>
>>>>>>>>>> of performance.<br>
>>>>><br>
>>>>> I've made additional testing, creating 2000 networks in<br>
>>>>> parallel (10 thread workers) for both drivers and<br>
>>>>> comparing results.<br>
>>>>><br>
>>>>> With mysqldb: 215.81 sec With mysql-connector: 88.66<br>
>>>>><br>
>>>>> ~2.4 times performance boost, ok? ;)<br>
>>>>><br>
>>>>> I think we should switch to that library *even* if we<br>
>>>>> forget about all the nasty deadlocks we experience now.<br>
>>>>><br>
>>>>>>>>>><br>
>>>>>>>>>> I've posted a Neutron spec for the switch to the<br>
>>>>>>>>>> new client in Juno at [1]. Ideally, switch is<br>
>>>>>>>>>> just a matter of several fixes to oslo.db that<br>
>>>>>>>>>> would enable full support for the new driver<br>
>>>>>>>>>> already supported by SQLAlchemy, plus<br>
>>>>>>>>>> 'connection' string modified in service<br>
>>>>>>>>>> configuration files, plus documentation updates<br>
>>>>>>>>>> to refer to the new official way to configure<br>
>>>>>>>>>> services for MySQL. The database code won't,<br>
>>>>>>>>>> ideally, require any major changes, though some<br>
>>>>>>>>>> adaptation for the new client library may be<br>
>>>>>>>>>> needed. That said, Neutron does not seem to<br>
>>>>>>>>>> require any changes, though it was revealed that<br>
>>>>>>>>>> there are some alembic migration rules in<br>
>>>>>>>>>> Keystone or Glance that need (trivial)<br>
>>>>>>>>>> modifications.<br>
>>>>>>>>>><br>
>>>>>>>>>> You can see how trivial the switch can be<br>
>>>>>>>>>> achieved for a service based on example for<br>
>>>>>>>>>> Neutron [2].<br>
>>>>>>>>>><br>
>>>>>>>>>> While this is a Neutron specific proposal, there<br>
>>>>>>>>>> is an obvious wish to switch to the new library<br>
>>>>>>>>>> globally throughout all the projects, to reduce<br>
>>>>>>>>>> devops burden, among other things. My vision is<br>
>>>>>>>>>> that, ideally, we switch all projects to the new<br>
>>>>>>>>>> library in Juno, though we still may leave<br>
>>>>>>>>>> several projects for K in case any issues arise,<br>
>>>>>>>>>> similar to the way projects switched to<br>
>>>>>>>>>> oslo.messaging during two cycles instead of one.<br>
>>>>>>>>>> Though looking at how easy Neutron can be<br>
>>>>>>>>>> switched to the new library, I wouldn't expect<br>
>>>>>>>>>> any issues that would postpone the switch till<br>
>>>>>>>>>> K.<br>
>>>>>>>>>><br>
>>>>>>>>>> It was mentioned in comments to the spec<br>
>>>>>>>>>> proposal that there were some discussions at the<br>
>>>>>>>>>> latest summit around possible switch in context<br>
>>>>>>>>>> of Nova that revealed some concerns, though they<br>
>>>>>>>>>> do not seem to be documented anywhere. So if you<br>
>>>>>>>>>> know anything about it, please comment.<br>
>>>>>>>>>><br>
>>>>>>>>>> So, we'd like to hear from other projects what's<br>
>>>>>>>>>> your take on that move, whether you see any<br>
>>>>>>>>>> issues or have concerns about it.<br>
>>>>>>>>>><br>
>>>>>>>>>> Thanks for your comments, /Ihar<br>
>>>>>>>>>><br>
>>>>>>>>>> [1]: <a href="https://review.openstack.org/#/c/104905/" target="_blank">https://review.openstack.org/#/c/104905/</a><br>
>>>>>>>>>> [2]: <a href="https://review.openstack.org/#/c/105209/" target="_blank">https://review.openstack.org/#/c/105209/</a><br>
>>>>>>>>>><br>
>>>>>>>>>> _______________________________________________<br>
>>>>>>>>>> OpenStack-dev mailing list<br>
>>>>>>>>>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>>>>>>>>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>>>>>>>>><br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>>>><br>
><br>
>>>>>>>>>><br>
>>>>>>>>>><br>
> _______________________________________________<br>
>>>>>>>> OpenStack-dev mailing list<br>
>>>>>>>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>>>>>>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>>><br>
><br>
>>>>>>>><br>
>>>>>>>><br>
> _______________________________________________<br>
>>>>>>> OpenStack-dev mailing list<br>
>>>>>>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>>>>>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>>><br>
><br>
>>>>>>><br>
>>>>>>><br>
> _______________________________________________<br>
>>>>>> OpenStack-dev mailing list<br>
>>>>>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>>>>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>>>><br>
>>>>>><br>
>>>>><br>
>>>>>><br>
><br>
>>>>>><br>
_______________________________________________ OpenStack-dev<br>
>>>>> mailing list <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>>>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>>>><br>
>>><br>
>>><br>
>>>>><br>
><br>
>>>>><br>
_______________________________________________<br>
>>> OpenStack-dev mailing list <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
>>><br>
>>><br>
>> _______________________________________________ OpenStack-dev<br>
>> mailing list <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
>><br>
><br>
> _______________________________________________ OpenStack-dev<br>
> mailing list <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
-----BEGIN PGP SIGNATURE-----<br>
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)<br>
Comment: Using GnuPG with Thunderbird - <a href="http://www.enigmail.net/" target="_blank">http://www.enigmail.net/</a><br>
<br>
</div></div>iQEcBAEBCgAGBQJTyQjjAAoJEC5aWaUY1u57560IAIwMf37adTOww1xtMkMFqBo2<br>
tXxvrJo+lhwF0B4CbEzaybNgCRd4N0UWpElDOkIU4Gqy4WLwp2Kf1+Qz8mUYAJEp<br>
tGo9wiQjlHSlUUNWTci4to1+dHztNomJYLkVD+EQsbGjC7quZhVGXYQPZV9rOR1c<br>
4V0h1vGvJoP4jLSE9KdAx8eQd81anAiKj+iPdWxiio2Xqcvjtt+qspoXzCpxRKlF<br>
6ghtihKhswmeJZ6G2KjxXJGZwXOkgigKICT4AwgnF14B8ZgRP9txOp2Ofy9GtUf0<br>
aYTPfRr5OyD80Cw+HNdOJw4haETNwTU9sBXI01Vv7WuqbDDVD1kIBVExgLMHw1g=<br>
=ubL5<br>
<div class="HOEnZb"><div class="h5">-----END PGP SIGNATURE-----<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br> - Gus
</div>