Open Stack

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 12/07/14 02:49, Jay Pipes wrote:
> On 07/11/2014 08:04 AM, Ihar Hrachyshka wrote: On 09/07/14 13:17,
> Ihar Hrachyshka wrote:
>>>> Hi all,
>>>> 
>>>> Multiple projects are suffering from db lock timeouts due to 
>>>> deadlocks deep in mysqldb library that we use to interact
>>>> with mysql servers. In essence, the problem is due to missing
>>>> eventlet support in mysqldb module, meaning when a db lock is
>>>> encountered, the library does not yield to the next green
>>>> thread, allowing other threads to eventually unlock the
>>>> grabbed lock, and instead it just blocks the main thread,
>>>> that eventually raises timeout exception (OperationalError).
>>>> 
>>>> The failed operation is not retried, leaving failing request
>>>> not served. In Nova, there is a special retry mechanism for
>>>> deadlocks, though I think it's more a hack than a proper
>>>> fix.
>>>> 
>>>> Neutron is one of the projects that suffer from those
>>>> timeout errors a lot. Partly it's due to lack of discipline
>>>> in how we do nested calls in l3_db and ml2_plugin code, but
>>>> that's not something to change in foreseeable future, so we
>>>> need to find another solution that is applicable for Juno.
>>>> Ideally, the solution should be applicable for Icehouse too
>>>> to allow distributors to resolve existing deadlocks without
>>>> waiting for Juno.
>>>> 
>>>> We've had several discussions and attempts to introduce a
>>>> solution to the problem. Thanks to oslo.db guys, we now have
>>>> more or less clear view on the cause of the failures and how
>>>> to easily fix them. The solution is to switch mysqldb to
>>>> something eventlet aware. The best candidate is probably
>>>> MySQL Connector module that is an official MySQL client for
>>>> Python and that shows some (preliminary) good results in
>>>> terms of performance.
> 
> I've made additional testing, creating 2000 networks in parallel
> (10 thread workers) for both drivers and comparing results.
> 
> With mysqldb: 215.81 sec With mysql-connector: 88.66
> 
> ~2.4 times performance boost, ok? ;)
> 
>> That really doesn't tell me much.
> 
>> Please remember that performance != scalability.

I actually agree. I need to spend more time preparing benchmarks (I'll
spend a part of my next week on this).

> 
>> If you showed the test/benchmark code, that would be great. You
>> need to

Here is a gist: https://gist.github.com/booxter/c4f3e743a2573ba7809f

>> run your benchmarks at varying levels of concurrency and varying
>> levels of read/write ratios for the workers. Otherwise it's like
>> looking at a a single dot of paint on a painting. Without looking
>> at the patterns of throughput (performance) and
>> concurrency/locking (scalability) with various levels of workers
>> and read/write ratios, you miss the whole picture.

Good point. I surely need to check read operations, and play with
multiple API/RPC workers.

Before I go with implementing my own tests, do you have more formal
requirements or suggestions on which data points we can be interested in?

> 
>> Another thing to ensure is that you are using real *processes*,
>> not threads, so that you actually simulate a real OpenStack
>> service like Nova or Neutron, which are multi-plexed, not
>> multi-threaded, and have a greenlet pool within each worker
>> process.

My test is an actual neutron client that issues full blown REST
requests to a local neutron server, in multiple thread workers. I
think this is ok, right?

> 
>> Best -jay
> 
> I think we should switch to that library *even* if we forget about
> all the nasty deadlocks we experience now.
> 
>>>> 
>>>> I've posted a Neutron spec for the switch to the new client
>>>> in Juno at [1]. Ideally, switch is just a matter of several
>>>> fixes to oslo.db that would enable full support for the new
>>>> driver already supported by SQLAlchemy, plus 'connection'
>>>> string modified in service configuration files, plus
>>>> documentation updates to refer to the new official way to
>>>> configure services for MySQL. The database code won't,
>>>> ideally, require any major changes, though some adaptation
>>>> for the new client library may be needed. That said, Neutron
>>>> does not seem to require any changes, though it was revealed
>>>> that there are some alembic migration rules in Keystone or 
>>>> Glance that need (trivial) modifications.
>>>> 
>>>> You can see how trivial the switch can be achieved for a
>>>> service based on example for Neutron [2].
>>>> 
>>>> While this is a Neutron specific proposal, there is an
>>>> obvious wish to switch to the new library globally throughout
>>>> all the projects, to reduce devops burden, among other
>>>> things. My vision is that, ideally, we switch all projects to
>>>> the new library in Juno, though we still may leave several
>>>> projects for K in case any issues arise, similar to the way
>>>> projects switched to oslo.messaging during two cycles instead
>>>> of one. Though looking at how easy Neutron can be switched to
>>>> the new library, I wouldn't expect any issues that would
>>>> postpone the switch till K.
>>>> 
>>>> It was mentioned in comments to the spec proposal that there
>>>> were some discussions at the latest summit around possible
>>>> switch in context of Nova that revealed some concerns, though
>>>> they do not seem to be documented anywhere. So if you know
>>>> anything about it, please comment.
>>>> 
>>>> So, we'd like to hear from other projects what's your take on
>>>> that move, whether you see any issues or have concerns about
>>>> it.
>>>> 
>>>> Thanks for your comments, /Ihar
>>>> 
>>>> [1]: https://review.openstack.org/#/c/104905/ [2]: 
>>>> https://review.openstack.org/#/c/105209/
>>>> 
>>>> _______________________________________________
>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org 
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>
>>
>>>> 
_______________________________________________
>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>> 
> 
> _______________________________________________ OpenStack-dev
> mailing list OpenStack-dev at lists.openstack.org 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCgAGBQJTwrUrAAoJEC5aWaUY1u57CFsH/2Yz8VZreSK1gFTLz8UxCSSV
3HGx/o3QlEomOWmSOkX4IfnE7XBouyPvIFoTwXdgrjwzQBW/iril+G2TA+B/RcER
+2ccZO9DkCKxCeAU4Tq/a7G92s3sLW+OgihxoIdpnEDdkMGa6XGDa/hLDrngaGyj
ow/DJeHDJVXd0xflTckMQusm8SfjLES/3yxW9A5rEH+f+4dn+bOf4SSQMdL44kLC
oTNQ4Fv6m0EOEg2JhliLbljFiJCfhDBXAHuoDT3zb+XGgIXG8zf4A8Q99pppM1Xd
snhtn5E5Fny+GdApBUb9JXE8uBo6E0EXMvZB2qPrwbk//GsxNtOLJxuWrmeffzM=
=Xprg
-----END PGP SIGNATURE-----