Open Stack

On Thu, Jan 29, 2015, at 07:03 PM, Angus Lees wrote:
> The other factor I haven't seen mentioned here is that our usual
> eventlet+mysqldb setup can deadlock rather easily(*), resulting in the
> entire python process sitting idle for 30s until the mysqldb deadlock
> timer
> goes off and raises an exception.  At this point (in nova at least), the
> request is usually retried and almost certainly succeeds.  In neutron (as
> another example), the requests weren't so easy to retry and code examples
> where this was noticed had to be explicitly restructured to defer work
> that
> might yield the greenlet.
> ie: it's a big cost on coding, and deployments, as well as just being
> plain
> incorrect code.
> 
> (*) eg: send simultaneous requests that modify the same resource.
> Eventually you'll have two greenlets within the same api server executing
> conflicting database operations.  Entire python process freezes.   (Note
> this is not a regular db transaction block, neither greenlet will ever
> exit
> their transactions without the deadlock timer).
> 
> Bug here: https://bugs.launchpad.net/oslo.db/+bug/1350149
> Test that reproduces the issue here:
> https://review.openstack.org/#/c/104436/
> 
> I'm dismayed that the conversation always skews towards discussing
> performance when the current choice isn't even correct yet, and that as a
> community we seem to be unwilling/unable to get behind what should be
> quite
> an obvious technical issue with straightforward solutions.

My impression of this thread is that we're reaching consensus that we
should move to PyMySQL, but that we want to make that move carefully
because we expect issues due to the changes in performance and context
switching behaviors. We've seen in the past that even slight changes in
some timings expose race conditions, so while raw performance isn't
critical the timing can be as tests run in the gate. Spending some time
on a migration & test plan, and making sure that people are ready to
help debug those sorts of issues, are good next steps.

The issue of moving off of eventlet can be addressed separately, after
the driver change.

So, do we need a cross-project spec to write down those details?

Doug

> 
> If we absolutely can't switch to another mysql driver, another option
> that
> was suggested recently (and passes the above test) is using
> eventlet.monkey_patch(MySQLdb=True).  I haven't done the investigation to
> find out why that isn't the default, or what the downsides are.  This
> obviously doesn't help us with other factors, like python3-ness either.
> 
>  - Gus
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev