Open Stack

On 7/9/14, 10:59 AM, Roman Podoliaka wrote:
> Hi all,
>
> Not sure what issues you are talking about, but I just replaced
> "mysql" with "mysql+mysqlconnector" in my db connection string  in
> neutron.conf and "neutron-db-manage upgrade head" worked like a charm
> for an empty schema.
>
> Ihar, could please elaborate on what changes to oslo.db are needed?
> (as an oslo.db developer I'm very interested in this part :) )
>
> Thanks,
> Roman
>
> On Wed, Jul 9, 2014 at 5:43 PM, Ihar Hrachyshka <ihrachys at redhat.com>
wrote:
> On 09/07/14 15:40, Sean Dague wrote:
> >>> On 07/09/2014 09:00 AM, Roman Podoliaka wrote:
> >>>> Hi Ihar,
> >>>>
> >>>> AFAIU, the switch is a matter of pip install + specifying the
> >>>> correct db URI in the config files. I'm not sure why you are
> >>>> filing a spec in Neutron project. IMHO, this has nothing to do
> >>>> with projects, but rather a purely deployment question. E.g.
> >>>> don't we have PostgreSQL+psycopg2 or MySQL+pymysql deployments of
> >>>> OpenStack right now?
> >>>>
> >>>> I think what you really want is to change the defaults we test in
> >>>> the gate, which is a different problem.
> >>>
> >>> Because this is really a *new* driver. As you can see by the
> >>> attempted run, it doesn't work with alembic given the definitions
> >>> that neutron has. So it's not like this is currently compatible
> >>> with OpenStack code.
>
> Well, to fix that, you just need to specify raise_on_warnings=False
> for connection (it's default for mysqldb but not mysql-connector).
> I've done it in devstack patch for now, but probably it belongs to

this is also semi-my fault as mysqlconnector apparently defaults this to
False now, but for some reason the SQLAlchemy mysqlconnector dialect is
flipping it to True (this dialect was contributed by MySQL-connector's
folks, so not sure why the inconsistency, perhaps they changed their minds)



> oslo.db.
>
> >>>
> >>>>
> >>>> Thanks, Roman
> >>>>
> >>>> On Wed, Jul 9, 2014 at 2:17 PM, Ihar Hrachyshka
> >>>> <ihrachys at redhat.com> wrote: Hi all,
> >>>>
> >>>> Multiple projects are suffering from db lock timeouts due to
> >>>> deadlocks deep in mysqldb library that we use to interact with
> >>>> mysql servers. In essence, the problem is due to missing eventlet
> >>>> support in mysqldb module, meaning when a db lock is encountered,
> >>>> the library does not yield to the next green thread, allowing
> >>>> other threads to eventually unlock the grabbed lock, and instead
> >>>> it just blocks the main thread, that eventually raises timeout
> >>>> exception (OperationalError).
> >>>>
> >>>> The failed operation is not retried, leaving failing request not
> >>>> served. In Nova, there is a special retry mechanism for
> >>>> deadlocks, though I think it's more a hack than a proper fix.
> >>>>
> >>>> Neutron is one of the projects that suffer from those timeout
> >>>> errors a lot. Partly it's due to lack of discipline in how we do
> >>>> nested calls in l3_db and ml2_plugin code, but that's not
> >>>> something to change in foreseeable future, so we need to find
> >>>> another solution that is applicable for Juno. Ideally, the
> >>>> solution should be applicable for Icehouse too to allow
> >>>> distributors to resolve existing deadlocks without waiting for
> >>>> Juno.
> >>>>
> >>>> We've had several discussions and attempts to introduce a
> >>>> solution to the problem. Thanks to oslo.db guys, we now have more
> >>>> or less clear view on the cause of the failures and how to easily
> >>>> fix them. The solution is to switch mysqldb to something eventlet
> >>>> aware. The best candidate is probably MySQL Connector module that
> >>>> is an official MySQL client for Python and that shows some
> >>>> (preliminary) good results in terms of performance.
> >>>>
> >>>> I've posted a Neutron spec for the switch to the new client in
> >>>> Juno at [1]. Ideally, switch is just a matter of several fixes to
> >>>> oslo.db that would enable full support for the new driver already
> >>>> supported by SQLAlchemy, plus 'connection' string modified in
> >>>> service configuration files, plus documentation updates to refer
> >>>> to the new official way to configure services for MySQL. The
> >>>> database code won't, ideally, require any major changes, though
> >>>> some adaptation for the new client library may be needed. That
> >>>> said, Neutron does not seem to require any changes, though it was
> >>>> revealed that there are some alembic migration rules in Keystone
> >>>> or Glance that need (trivial) modifications.
> >>>>
> >>>> You can see how trivial the switch can be achieved for a service
> >>>> based on example for Neutron [2].
> >>>>
> >>>> While this is a Neutron specific proposal, there is an obvious
> >>>> wish to switch to the new library globally throughout all the
> >>>> projects, to reduce devops burden, among other things. My vision
> >>>> is that, ideally, we switch all projects to the new library in
> >>>> Juno, though we still may leave several projects for K in case
> >>>> any issues arise, similar to the way projects switched to
> >>>> oslo.messaging during two cycles instead of one. Though looking
> >>>> at how easy Neutron can be switched to the new library, I
> >>>> wouldn't expect any issues that would postpone the switch till
> >>>> K.
> >>>>
> >>>> It was mentioned in comments to the spec proposal that there were
> >>>> some discussions at the latest summit around possible switch in
> >>>> context of Nova that revealed some concerns, though they do not
> >>>> seem to be documented anywhere. So if you know anything about it,
> >>>> please comment.
> >>>>
> >>>> So, we'd like to hear from other projects what's your take on
> >>>> that move, whether you see any issues or have concerns about it.
> >>>>
> >>>> Thanks for your comments, /Ihar
> >>>>
> >>>> [1]: https://review.openstack.org/#/c/104905/ [2]:
> >>>> https://review.openstack.org/#/c/105209/
> >>>>>
> >>>>> _______________________________________________ OpenStack-dev
> >>>>> mailing list OpenStack-dev at lists.openstack.org
> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>
> >>>>
> >>>>>
> _______________________________________________
> >>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>
> >>>
> >>>>
> >>>
> >>>
> >>> _______________________________________________ OpenStack-dev
> >>> mailing list OpenStack-dev at lists.openstack.org
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140711/ad3cd7f7/attachment.html>