Open Stack

On 07/11/2014 08:04 AM, Ihar Hrachyshka wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> On 09/07/14 13:17, Ihar Hrachyshka wrote:
>> Hi all,
>>
>> Multiple projects are suffering from db lock timeouts due to
>> deadlocks deep in mysqldb library that we use to interact with
>> mysql servers. In essence, the problem is due to missing eventlet
>> support in mysqldb module, meaning when a db lock is encountered,
>> the library does not yield to the next green thread, allowing other
>> threads to eventually unlock the grabbed lock, and instead it just
>> blocks the main thread, that eventually raises timeout exception
>> (OperationalError).
>>
>> The failed operation is not retried, leaving failing request not
>> served. In Nova, there is a special retry mechanism for deadlocks,
>> though I think it's more a hack than a proper fix.
>>
>> Neutron is one of the projects that suffer from those timeout
>> errors a lot. Partly it's due to lack of discipline in how we do
>> nested calls in l3_db and ml2_plugin code, but that's not something
>> to change in foreseeable future, so we need to find another
>> solution that is applicable for Juno. Ideally, the solution should
>> be applicable for Icehouse too to allow distributors to resolve
>> existing deadlocks without waiting for Juno.
>>
>> We've had several discussions and attempts to introduce a solution
>> to the problem. Thanks to oslo.db guys, we now have more or less
>> clear view on the cause of the failures and how to easily fix them.
>> The solution is to switch mysqldb to something eventlet aware. The
>> best candidate is probably MySQL Connector module that is an
>> official MySQL client for Python and that shows some (preliminary)
>> good results in terms of performance.
>
> I've made additional testing, creating 2000 networks in parallel (10
> thread workers) for both drivers and comparing results.
>
> With mysqldb: 215.81 sec
> With mysql-connector: 88.66
>
> ~2.4 times performance boost, ok? ;)

That really doesn't tell me much.

Please remember that performance != scalability.

If you showed the test/benchmark code, that would be great. You need to 
run your benchmarks at varying levels of concurrency and varying levels 
of read/write ratios for the workers. Otherwise it's like looking at a a 
single dot of paint on a painting. Without looking at the patterns of 
throughput (performance) and concurrency/locking (scalability) with 
various levels of workers and read/write ratios, you miss the whole picture.

Another thing to ensure is that you are using real *processes*, not 
threads, so that you actually simulate a real OpenStack service like 
Nova or Neutron, which are multi-plexed, not multi-threaded, and have a 
greenlet pool within each worker process.

Best
-jay

> I think we should switch to that library *even* if we forget about all
> the nasty deadlocks we experience now.
>
>>
>> I've posted a Neutron spec for the switch to the new client in Juno
>> at [1]. Ideally, switch is just a matter of several fixes to
>> oslo.db that would enable full support for the new driver already
>> supported by SQLAlchemy, plus 'connection' string modified in
>> service configuration files, plus documentation updates to refer to
>> the new official way to configure services for MySQL. The database
>> code won't, ideally, require any major changes, though some
>> adaptation for the new client library may be needed. That said,
>> Neutron does not seem to require any changes, though it was
>> revealed that there are some alembic migration rules in Keystone or
>> Glance that need (trivial) modifications.
>>
>> You can see how trivial the switch can be achieved for a service
>> based on example for Neutron [2].
>>
>> While this is a Neutron specific proposal, there is an obvious wish
>> to switch to the new library globally throughout all the projects,
>> to reduce devops burden, among other things. My vision is that,
>> ideally, we switch all projects to the new library in Juno, though
>> we still may leave several projects for K in case any issues arise,
>> similar to the way projects switched to oslo.messaging during two
>> cycles instead of one. Though looking at how easy Neutron can be
>> switched to the new library, I wouldn't expect any issues that
>> would postpone the switch till K.
>>
>> It was mentioned in comments to the spec proposal that there were
>> some discussions at the latest summit around possible switch in
>> context of Nova that revealed some concerns, though they do not
>> seem to be documented anywhere. So if you know anything about it,
>> please comment.
>>
>> So, we'd like to hear from other projects what's your take on that
>> move, whether you see any issues or have concerns about it.
>>
>> Thanks for your comments, /Ihar
>>
>> [1]: https://review.openstack.org/#/c/104905/ [2]:
>> https://review.openstack.org/#/c/105209/
>>
>> _______________________________________________ OpenStack-dev
>> mailing list OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQEcBAEBCgAGBQJTv9LHAAoJEC5aWaUY1u57d2cIAIAthLuM6qxN9fVjPwoICEae
> oSOLvaDNPpZ+xBBqKI+2l5aFiBXSkHzgCfWGHEZB4e+5odAzt8r3Dg5eG/hwckGt
> iZLPGLxcmvD5K0cRoSSPWkPC4KkOwKw0yQHl/JQarDcHQlLgO64jx3bzlB1LDxRu
> R/Bvqo1SBo8g/cupWyxJXNViu9z7zAlvcHLRg4j/AfNTsTDZRrSgbMF2/gLTMvN2
> FPtkjBvZq++zOva5G5/TySr1b3QRBFCG0uetVbcVF//90XOw+O++rUiDW1v7vkA9
> OS2sCIXmx1i8kt9yuvs0h11MS8qfX9rSXREJXyPq6NDmePdQdKFsozMdTmqaDfU=
> =JfiC
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>