Open Stack

On 07/09/2014 09:00 AM, Roman Podoliaka wrote:
> Hi Ihar,
> 
> AFAIU, the switch is a matter of pip install + specifying the correct
> db URI in the config files. I'm not sure why you are filing a spec in
> Neutron project. IMHO, this has nothing to do with projects, but
> rather a purely deployment question. E.g. don't we have
> PostgreSQL+psycopg2 or MySQL+pymysql deployments of OpenStack right
> now?
> 
> I think what you really want is to change the defaults we test in the
> gate, which is a different problem.

Because this is really a *new* driver. As you can see by the attempted
run, it doesn't work with alembic given the definitions that neutron
has. So it's not like this is currently compatible with OpenStack code.

> 
> Thanks,
> Roman
> 
> On Wed, Jul 9, 2014 at 2:17 PM, Ihar Hrachyshka <ihrachys at redhat.com> wrote:
> Hi all,
> 
> Multiple projects are suffering from db lock timeouts due to deadlocks
> deep in mysqldb library that we use to interact with mysql servers. In
> essence, the problem is due to missing eventlet support in mysqldb
> module, meaning when a db lock is encountered, the library does not
> yield to the next green thread, allowing other threads to eventually
> unlock the grabbed lock, and instead it just blocks the main thread,
> that eventually raises timeout exception (OperationalError).
> 
> The failed operation is not retried, leaving failing request not
> served. In Nova, there is a special retry mechanism for deadlocks,
> though I think it's more a hack than a proper fix.
> 
> Neutron is one of the projects that suffer from those timeout errors a
> lot. Partly it's due to lack of discipline in how we do nested calls
> in l3_db and ml2_plugin code, but that's not something to change in
> foreseeable future, so we need to find another solution that is
> applicable for Juno. Ideally, the solution should be applicable for
> Icehouse too to allow distributors to resolve existing deadlocks
> without waiting for Juno.
> 
> We've had several discussions and attempts to introduce a solution to
> the problem. Thanks to oslo.db guys, we now have more or less clear
> view on the cause of the failures and how to easily fix them. The
> solution is to switch mysqldb to something eventlet aware. The best
> candidate is probably MySQL Connector module that is an official MySQL
> client for Python and that shows some (preliminary) good results in
> terms of performance.
> 
> I've posted a Neutron spec for the switch to the new client in Juno at
> [1]. Ideally, switch is just a matter of several fixes to oslo.db that
> would enable full support for the new driver already supported by
> SQLAlchemy, plus 'connection' string modified in service configuration
> files, plus documentation updates to refer to the new official way to
> configure services for MySQL. The database code won't, ideally,
> require any major changes, though some adaptation for the new client
> library may be needed. That said, Neutron does not seem to require any
> changes, though it was revealed that there are some alembic migration
> rules in Keystone or Glance that need (trivial) modifications.
> 
> You can see how trivial the switch can be achieved for a service based
> on example for Neutron [2].
> 
> While this is a Neutron specific proposal, there is an obvious wish to
> switch to the new library globally throughout all the projects, to
> reduce devops burden, among other things. My vision is that, ideally,
> we switch all projects to the new library in Juno, though we still may
> leave several projects for K in case any issues arise, similar to the
> way projects switched to oslo.messaging during two cycles instead of
> one. Though looking at how easy Neutron can be switched to the new
> library, I wouldn't expect any issues that would postpone the switch
> till K.
> 
> It was mentioned in comments to the spec proposal that there were some
> discussions at the latest summit around possible switch in context of
> Nova that revealed some concerns, though they do not seem to be
> documented anywhere. So if you know anything about it, please comment.
> 
> So, we'd like to hear from other projects what's your take on that
> move, whether you see any issues or have concerns about it.
> 
> Thanks for your comments,
> /Ihar
> 
> [1]: https://review.openstack.org/#/c/104905/
> [2]: https://review.openstack.org/#/c/105209/
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

-- 
Sean Dague
http://dague.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140709/d44176f2/attachment.pgp>