Open Stack

Excerpts from jeblair's message of 2014-03-26 13:30:28 -0700:
> Hi,
> 
> I recently started looking into running the unit tests with MySQL.  I
> pushed the start of a change here to do that:
> 
> https://review.openstack.org/#/c/82872/
> 
> In particular, that change implements something we've wanted to do for a
> long time elsewhere in OpenStack -- create a dedicated MySQL database
> for each test so that tests can safely run in parallel.  However, that
> has uncovered the following issues:
> 
>  * The tests do not use alembic, except for the migration test. This
>    could lead to variations in what the unit tests are testing against
>    as compared to production.
> 
>  * The model does not specify an engine for MySQL (this is a difference
>    between the model and the alembic migrations). So when the unit tests
>    are run on MySQL, they may end up on MyISAM (they do in the gate).
> 

Why MyISAM? The default has been InnoDB since MySQL 5.5.

>  * The two errors that show up in the test results for my patch relate
>    to how timestamps are stored differently between MySQL and SQLite.
> 

Isn't this a bug in sqlalchemy? I'd expect any data type that is a common
name in the two databases (and thus the same in the model definition)
to produce the same data in the python representation.

>  * The errors that you can not see in the test report from Jenkins but
>    would if you ran in an environment with InnoDB as the default engine
>    for MySQL are several instances of foreign key constraint errors.
> 

Side note: IMHO: Just drop the FK constraints if you want the app to
be scalable.  FK's just add extra reads for every write, when you could
be doing that verification / orphan clean up offline later on.

> I think it is desirable to run the unit tests with MySQL largely because
> I think we should be testing it in the way we intend to use it in
> production.  Note especially the last point above -- our unit tests have
> significant problems with MySQL.  That suggests one of two things to me:
> either the tests aren't testing real behavior, or our system is actually
> broken in production.  Neither one seems like a good situation.
> 

Wouldn't integration tests be the place to find production vs. code path
issues? Tying unit tests to the creation of mysql databases introduces a
really big piece of machinery into the test fixture setup, which you
want to be fast and lean so developers can iterate on it while working.

> My inclination would be to use MySQL for unit tests, use alembic to
> create the schema, and fix the errors currently uncovered by this.
> 
> It's not going to be a trivial change, but I think it's worth doing.
> Before embarking on it, I'd like to see if we can get consensus.  We can
> talk about it here or possibly at the meeting tomorrow.
> 
> Ruslan has commented in that review with the following:
> 
> > But, I think that we should consider work done in major OpenStack
> > projects. They don't use real MySQL to run unit-tests. Here is what
> > the do (or plan to do):
> >
> > * Run unit-tests on SQLite. DB schema is created from SQLA metadata
> > * Run DB migration tests on MySQL and Postgres
> > * Run tests to compare SQLA metadata and DB migrations to make sure
> >   that they're in sync. Here is WIP patch
> >   https://review.openstack.org/#/c/
> >
> > I'm not opposed to this direction, I just want to make sure we're
> > aligned with the common practices in OpenStack
> 
> (That last link doesn't seem to be correct.)
> 
> That's a good point, but I personally find testing as close to
> production as possible desirable -- MySQL and SQLite are not compatible
> so we shouldn't pretend that they are.  If we prove this is a viable
> strategy, then I think we should recommend OpenStack projects adopt it
> as well.
> 

I just wonder if SQLAlchemy is intended to "smooth out" the differences
between the two in such a way where you can at least be confident that
your code is doing what you think it is.