Open Stack

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 14/07/14 17:03, Ihar Hrachyshka wrote:
> On 14/07/14 15:54, Clark Boylan wrote:
>> On Sun, Jul 13, 2014 at 9:20 AM, Ihar Hrachyshka 
>> <ihrachys at redhat.com> wrote: On 11/07/14 19:20, Clark Boylan 
>> wrote:
>>>>> Before we get too far ahead of ourselves mysql-connector
>>>>> is not hosted on pypi. Instead it is an external package
>>>>> link. We recently managed to remove all packages that are
>>>>> hosted as external package links from openstack and will
>>>>> not add new ones in. Before we can use mysql-connector in
>>>>> the gate oracle will need to publish mysql-connector on
>>>>> pypi properly.
> 
>> There is misunderstanding in our community on how we deploy db 
>> client modules. No project actually depends on any of them. We 
>> assume deployers will install the proper one and configure 
>> 'connection' string to use it. In case of devstack, we install
>> the appropriate package from distribution packages, not pip.
> 
>>> Correct, but for all of the other test suites (unittests) we 
>>> install the db clients via pip because tox runs them and 
>>> virtualenvs allowing site packages cause too many problems. See
>>>  
>>> https://git.openstack.org/cgit/openstack/nova/tree/test-requirements.txt#n8.
>>>
>>>
>
>>> 
So we do actually depend on these things being pip installable.
>>> Basically this allows devs to run `tox` and it works.
> 
> Roger that, and thanks for clarification. I'm trying to reach the 
> author and the maintainer of mysqlconnector-python to see whether
> I'll be able to convince him to publish the packages on
> pypi.python.org.
> 

I've reached the maintainer of the module, he told me he is currently
working on uploading releases directly to pypi.python.org.

> 
>>> I would argue that we should have devstack install via pip too 
>>> for consistency, but that is a different issue (it is already 
>>> installing all of the other python dependencies this way so
>>> why special case?).
> 
>> What we do is recommending a module for our users in our 
>> documentation.
> 
>> That said, I assume the gate is a non-issue. Correct?
> 
>>>>> 
>>>>> That said there is at least one other pure python 
>>>>> alternative, PyMySQL. PyMySQL supports py3k and pypy. We 
>>>>> should look at using PyMySQL instead if we want to start
>>>>> with a reasonable path to getting this in the gate.
> 
>> MySQL Connector supports py3k too (not sure about pypy though).
> 
>>>>> 
>>>>> Clark
>>>>> 
>>>>> On Fri, Jul 11, 2014 at 10:07 AM, Miguel Angel Ajo Pelayo 
>>>>> <mangelajo at redhat.com> wrote:
>>>>>> +1 here too,
>>>>>> 
>>>>>> Amazed with the performance gains, x2.4 seems a lot, and 
>>>>>> we'd get rid of deadlocks.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> ----- Original Message -----
>>>>>>> +1
>>>>>>> 
>>>>>>> I'm pretty excited about the possibilities here.  I've 
>>>>>>> had this mysqldb/eventlet contention in the back of my 
>>>>>>> mind for some time now. I'm glad to see some work
>>>>>>> being done in this area.
>>>>>>> 
>>>>>>> Carl
>>>>>>> 
>>>>>>> On Fri, Jul 11, 2014 at 7:04 AM, Ihar Hrachyshka 
>>>>>>> <ihrachys at redhat.com> wrote:
>>>>> On 09/07/14 13:17, Ihar Hrachyshka wrote:
>>>>>>>>>> Hi all,
>>>>>>>>>> 
>>>>>>>>>> Multiple projects are suffering from db lock 
>>>>>>>>>> timeouts due to deadlocks deep in mysqldb
>>>>>>>>>> library that we use to interact with mysql
>>>>>>>>>> servers. In essence, the problem is due to
>>>>>>>>>> missing eventlet support in mysqldb module,
>>>>>>>>>> meaning when a db lock is encountered, the
>>>>>>>>>> library does not yield to the next green thread,
>>>>>>>>>> allowing other threads to eventually unlock the
>>>>>>>>>> grabbed lock, and instead it just blocks the main
>>>>>>>>>> thread, that eventually raises timeout exception
>>>>>>>>>> (OperationalError).
>>>>>>>>>> 
>>>>>>>>>> The failed operation is not retried, leaving 
>>>>>>>>>> failing request not served. In Nova, there is a 
>>>>>>>>>> special retry mechanism for deadlocks, though I 
>>>>>>>>>> think it's more a hack than a proper fix.
>>>>>>>>>> 
>>>>>>>>>> Neutron is one of the projects that suffer from 
>>>>>>>>>> those timeout errors a lot. Partly it's due to
>>>>>>>>>> lack of discipline in how we do nested calls in
>>>>>>>>>> l3_db and ml2_plugin code, but that's not
>>>>>>>>>> something to change in foreseeable future, so we
>>>>>>>>>> need to find another solution that is applicable
>>>>>>>>>> for Juno. Ideally, the solution should be
>>>>>>>>>> applicable for Icehouse too to allow distributors
>>>>>>>>>> to resolve existing deadlocks without waiting for
>>>>>>>>>> Juno.
>>>>>>>>>> 
>>>>>>>>>> We've had several discussions and attempts to 
>>>>>>>>>> introduce a solution to the problem. Thanks to 
>>>>>>>>>> oslo.db guys, we now have more or less clear
>>>>>>>>>> view on the cause of the failures and how to
>>>>>>>>>> easily fix them. The solution is to switch
>>>>>>>>>> mysqldb to something eventlet aware. The best
>>>>>>>>>> candidate is probably MySQL Connector module that
>>>>>>>>>> is an official MySQL client for Python and that
>>>>>>>>>> shows some (preliminary) good results in terms
>>>>>>>>>> of performance.
>>>>> 
>>>>> I've made additional testing, creating 2000 networks in 
>>>>> parallel (10 thread workers) for both drivers and
>>>>> comparing results.
>>>>> 
>>>>> With mysqldb: 215.81 sec With mysql-connector: 88.66
>>>>> 
>>>>> ~2.4 times performance boost, ok? ;)
>>>>> 
>>>>> I think we should switch to that library *even* if we
>>>>> forget about all the nasty deadlocks we experience now.
>>>>> 
>>>>>>>>>> 
>>>>>>>>>> I've posted a Neutron spec for the switch to the 
>>>>>>>>>> new client in Juno at [1]. Ideally, switch is
>>>>>>>>>> just a matter of several fixes to oslo.db that
>>>>>>>>>> would enable full support for the new driver
>>>>>>>>>> already supported by SQLAlchemy, plus
>>>>>>>>>> 'connection' string modified in service
>>>>>>>>>> configuration files, plus documentation updates
>>>>>>>>>> to refer to the new official way to configure
>>>>>>>>>> services for MySQL. The database code won't,
>>>>>>>>>> ideally, require any major changes, though some
>>>>>>>>>> adaptation for the new client library may be
>>>>>>>>>> needed. That said, Neutron does not seem to 
>>>>>>>>>> require any changes, though it was revealed that 
>>>>>>>>>> there are some alembic migration rules in
>>>>>>>>>> Keystone or Glance that need (trivial)
>>>>>>>>>> modifications.
>>>>>>>>>> 
>>>>>>>>>> You can see how trivial the switch can be
>>>>>>>>>> achieved for a service based on example for
>>>>>>>>>> Neutron [2].
>>>>>>>>>> 
>>>>>>>>>> While this is a Neutron specific proposal, there
>>>>>>>>>> is an obvious wish to switch to the new library 
>>>>>>>>>> globally throughout all the projects, to reduce 
>>>>>>>>>> devops burden, among other things. My vision is 
>>>>>>>>>> that, ideally, we switch all projects to the new 
>>>>>>>>>> library in Juno, though we still may leave
>>>>>>>>>> several projects for K in case any issues arise,
>>>>>>>>>> similar to the way projects switched to
>>>>>>>>>> oslo.messaging during two cycles instead of one.
>>>>>>>>>> Though looking at how easy Neutron can be
>>>>>>>>>> switched to the new library, I wouldn't expect
>>>>>>>>>> any issues that would postpone the switch till
>>>>>>>>>> K.
>>>>>>>>>> 
>>>>>>>>>> It was mentioned in comments to the spec
>>>>>>>>>> proposal that there were some discussions at the
>>>>>>>>>> latest summit around possible switch in context
>>>>>>>>>> of Nova that revealed some concerns, though they
>>>>>>>>>> do not seem to be documented anywhere. So if you
>>>>>>>>>> know anything about it, please comment.
>>>>>>>>>> 
>>>>>>>>>> So, we'd like to hear from other projects what's 
>>>>>>>>>> your take on that move, whether you see any
>>>>>>>>>> issues or have concerns about it.
>>>>>>>>>> 
>>>>>>>>>> Thanks for your comments, /Ihar
>>>>>>>>>> 
>>>>>>>>>> [1]: https://review.openstack.org/#/c/104905/
>>>>>>>>>> [2]: https://review.openstack.org/#/c/105209/
>>>>>>>>>> 
>>>>>>>>>> _______________________________________________ 
>>>>>>>>>> OpenStack-dev mailing list 
>>>>>>>>>> OpenStack-dev at lists.openstack.org 
>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>>
>
>>>>>>>>>> 
>>>>>>>>>> 
> _______________________________________________
>>>>>>>> OpenStack-dev mailing list 
>>>>>>>> OpenStack-dev at lists.openstack.org 
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>>
>>>>>>>>
>
>>>>>>>> 
>>>>>>>> 
> _______________________________________________
>>>>>>> OpenStack-dev mailing list 
>>>>>>> OpenStack-dev at lists.openstack.org 
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>
>>>>>>> 
>>>>>>> 
> _______________________________________________
>>>>>> OpenStack-dev mailing list 
>>>>>> OpenStack-dev at lists.openstack.org 
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>>
>>>>>
>>>>>>
>
>>>>>> 
_______________________________________________ OpenStack-dev
>>>>> mailing list OpenStack-dev at lists.openstack.org 
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>
>>>
>>>>>
>
>>>>> 
_______________________________________________
>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org 
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>>> 
>>> 
>> _______________________________________________ OpenStack-dev 
>> mailing list OpenStack-dev at lists.openstack.org 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>> 
> 
> _______________________________________________ OpenStack-dev
> mailing list OpenStack-dev at lists.openstack.org 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCgAGBQJTyQjjAAoJEC5aWaUY1u57560IAIwMf37adTOww1xtMkMFqBo2
tXxvrJo+lhwF0B4CbEzaybNgCRd4N0UWpElDOkIU4Gqy4WLwp2Kf1+Qz8mUYAJEp
tGo9wiQjlHSlUUNWTci4to1+dHztNomJYLkVD+EQsbGjC7quZhVGXYQPZV9rOR1c
4V0h1vGvJoP4jLSE9KdAx8eQd81anAiKj+iPdWxiio2Xqcvjtt+qspoXzCpxRKlF
6ghtihKhswmeJZ6G2KjxXJGZwXOkgigKICT4AwgnF14B8ZgRP9txOp2Ofy9GtUf0
aYTPfRr5OyD80Cw+HNdOJw4haETNwTU9sBXI01Vv7WuqbDDVD1kIBVExgLMHw1g=
=ubL5
-----END PGP SIGNATURE-----