Open Stack

On Wed, 20 Aug 2014 05:03:51 PM Clark Boylan wrote:
> On Mon, Aug 18, 2014, at 01:59 AM, Ihar Hrachyshka wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA512
> > 
> > On 17/08/14 02:09, Angus Lees wrote:
> > > On 16 Aug 2014 06:09, "Doug Hellmann" <doug at doughellmann.com
> > > 
> > > <mailto:doug at doughellmann.com>> wrote:
> > >> On Aug 15, 2014, at 9:29 AM, Ihar Hrachyshka
> > >> <ihrachys at redhat.com
> > > 
> > > <mailto:ihrachys at redhat.com>> wrote:
> > >>> Signed PGP part Some updates on the matter:
> > >>> 
> > >>> - oslo-spec was approved with narrowed scope which is now
> > >>> 'enabled mysqlconnector as an alternative in gate' instead of
> > >>> 'switch the default db driver to mysqlconnector'. We'll revisit
> > >>> the switch part the next cycle once we have the new driver
> > >>> running in gate and real benchmarking is heavy-lifted.
> > >>> 
> > >>> - there are several patches that are needed to make devstack
> > >>> and tempest passing deployment and testing. Those are collected
> > >>> under the hood of: https://review.openstack.org/#/c/114207/ Not
> > >>> much of them.
> > >>> 
> > >>> - we'll need a new oslo.db release to bump versions (this is
> > >>> needed to set raise_on_warnings=False for the new driver, which
> > >>> was incorrectly set to True in sqlalchemy till very recently).
> > >>> This is expected to be released this month (as per Roman
> > >>> Podoliaka).
> > >> 
> > >> This release is currently blocked on landing some changes in
> > >> projects
> > > 
> > > using the library so they don?t break when the new version starts
> > > using different exception classes. We?re tracking that work in
> > > https://etherpad.openstack.org/p/sqla_exceptions_caught
> > > 
> > >> It looks like we?re down to 2 patches, one for cinder
> > > 
> > > (https://review.openstack.org/#/c/111760/) and one for glance
> > > (https://review.openstack.org/#/c/109655). Roman, can you verify
> > > that those are the only two projects that need changes for the
> > > exception issue?
> > > 
> > >>> - once the corresponding patch for sqlalchemy-migrate is
> > >>> merged, we'll also need a new version released for this.
> > > 
> > > So we're going for a new version of sqlalchemy?  (We have a
> > > separate workaround for raise_on_warnings that doesn't require the
> > > new sqlalchemy release if this brings too many other issues)
> > 
> > Wrong. We're going for a new version of *sqlalchemy-migrate*. Which is
> > the code that we inherited from Mike and currently track in stackforge.
> > 
> > >>> - on PyPI side, no news for now. The last time I've heard from
> > >>> Geert (the maintainer of MySQL Connector for Python), he was
> > >>> working on this. I suspect there are some legal considerations
> > >>> running inside Oracle. I'll update once I know more about
> > >>> that.
> > >> 
> > >> If we don?t have the new package on PyPI, how do we plan to
> > >> include it
> > > 
> > > in the gate? Are there options to allow an exception, or to make
> > > the mirroring software download it anyway?
> > > 
> > > We can test via devstack without waiting for pypi, since devstack
> > > will install via rpms/debs.
> > 
> > I expect that it will be settled. I have no indication that the issue
> > is unsolvable, it will just take a bit more time than we're accustomed
> > to. :)
> > 
> > At the moment, we install MySQLdb from distro packages for devstack.
> > Same applies to new driver. It will be still great to see the package
> > published on PyPI so that we can track its version requirements
> > instead of relying on distros to package it properly. But I don't see
> > it as a blocker.
> > 
> > Also, we will probably be able to run with other drivers supported by
> > SQLAlchemy once all the work is done.
> 
> So I got bored last night and decided to take a stab at making PyMySQL
> work since I was a proponent of it earlier. Thankfully it did just
> mostly work like I thought it would.
> https://review.openstack.org/#/c/115495/ is the WIP devstack change to
> test this out.

Thanks!

> Postgres tests fail because it was applying the pymysql driver to the
> postgres connection string. We can clean this up later in devstack
> and/or devstack-gate depending on how we need to apply this stuff.
> Bashate failed because I had to "monkeypatch" in a fix for a ceilometer
> issue loading sqlalchemy drivers. The tempest neutron full job fails on
> one test occasionally. Not sure yet if that is normal neutron full
> failure mode or if a new thing from PyMySQL. The regular tempest job
> passes just fine.
> 
> There are also some DB related errors in the logs that will need to be
> cleaned up but overall it just works. So I would like to repropose that
> we stop focusing all of this effort on the hard thing and use the easy
> thing if it meets our needs. We can continue to make alternatives work,
> but that is a different problem that we can solve at a different pace. I
> am not sure how to test the neutron thing that Gus was running into
> though so we should also check that really quickly.

TL;DR: pymysql passes my test case.
I'm perfectly happy to move towards using mysql+pymysql in gate tests.  (The 
various changes I've been submitting are to support _any_ non-default driver).

If anyone cares, my test case is in https://review.openstack.org/#/c/104436/
Unfortunately it requires a small amount of hacking up the relevant utils.py 
functions to test with an arbitrary sqlalchemy connect string.  I had some 
clumsy attempts at making that easier, but they were rejected in favour of a 
far better solution that still hasn't landed yet :/

-- 
 - Gus